Add tilepyramids to PyramidScheme #49

meggart · 2024-08-01T09:28:14Z

This adds abstraction layers to treat any Pyramid as a TileProvider and also adds a Pyramid constructor to lazily expose a TileProvider as a Pyramid.

codecov · 2024-08-01T10:13:39Z

Codecov Report

Attention: Patch coverage is 40.16393% with 73 lines in your changes missing coverage. Please review.

Project coverage is 57.88%. Comparing base (4bb73dd) to head (0f5ac9c).

Files	Patch %	Lines
src/pyramidprovider.jl	0.00%	28 Missing ⚠️
ext/PyramidSchemeTylerExt.jl	0.00%	23 Missing ⚠️
src/tilepyramids.jl	72.05%	19 Missing ⚠️
src/PyramidScheme.jl	0.00%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main      #49      +/-   ##
==========================================
- Coverage   66.03%   57.88%   -8.16%     
==========================================
  Files           3        6       +3     
  Lines         265      387     +122     
==========================================
+ Hits          175      224      +49     
- Misses         90      163      +73

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

felixcremer · 2024-08-01T10:18:28Z

We should also add some tests for this functionality. I took care of the compat entries already.

src/tilepyramids.jl

felixcremer · 2024-08-01T12:05:46Z

src/tilepyramids.jl

+    if mode === :band
+        return MapTileDiskArray{rgbeltype(et),3,typeof(prov)}(prov, zoom, size(testtile, 1), nband)
+    elseif mode === :rgb
+        return MapTileDiskArray{et,2,typeof(prov)}(prov, zoom, size(testtile, 1), nband)


Shouldn't the number of bands be one in the case that the RGB info is in the color element type?

Well there are no bands in this case, since the resulting array is 2d, and the field should never be accessed

felixcremer · 2024-08-01T12:09:49Z

src/tilepyramids.jl

+    y1, y2 = first(ex1.Y), last(ex2.Y)
+    stepx = (x2 - x1) / npix
+    stepy = (y2 - y1) / npix
+    x = X(range(x1 + stepx / 2, x2 - stepx / 2, length=npix))


Should we add here that these are intervals? Because most likely in the tiles the value of a pixel should represent the average over the whole pixel area and not a point measurement.

Yes, this would indeed be better, I will look into it

felixcremer · 2024-08-01T15:18:35Z

The eachchunk result of a Pyramid from the Openstreetmap providers seems wrong for me.
This is only for the base layer and I used the pyramid that is in the test that I am going to push.

julia> size(eachchunk(pyr))
(1, 5, 134217728)

julia> eachchunk(pyr)[1,1,1]
(1:3, 1:33333333, 1:1)

this can still get some tests for the chunking structure.

felixcremer · 2024-08-02T06:52:55Z

plotting of a pyramid from a provider does not work.

using GLMakie, PyramidScheme, TileProviders
julia> prov = TileProviders.OpenStreetMap(); pyr = Pyramid(prov)
julia> plot(pyr, colorbar=false)
ERROR: MethodError: no method matching defaultlimits(::Tuple{String, String}, ::typeof(identity))

Closest candidates are:
  defaultlimits(::Tuple{Any, Any}, ::Any, ::Any)
   @ Makie ~/.julia/packages/Makie/qMluh/src/makielayout/blocks/axis.jl:1418
  defaultlimits(::Tuple{Nothing, Nothing}, ::Any)
   @ Makie ~/.julia/packages/Makie/qMluh/src/makielayout/blocks/axis.jl:1429
  defaultlimits(::Nothing, ::Any)
   @ Makie ~/.julia/packages/Makie/qMluh/src/makielayout/blocks/axis.jl:1424
  ...

Stacktrace:
  [1] defaultlimits(userlimits::Tuple{Tuple{String, String}, Tuple{Float64, Float64}}, xscale::Function, yscale::Function)
    @ Makie ~/.julia/packages/Makie/qMluh/src/makielayout/blocks/axis.jl:1419
  [2] initialize_block!(ax::Axis; palette::Nothing)
    @ Makie ~/.julia/packages/Makie/qMluh/src/makielayout/blocks/axis.jl:158
  [3] initialize_block!(ax::Axis)
    @ Makie ~/.julia/packages/Makie/qMluh/src/makielayout/blocks/axis.jl:150
  [4] _block(T::Type{…}, fig_or_scene::Figure, args::Vector{…}, kwdict::Dict{…}, bbox::Nothing; kwdict_complete::Bool)
    @ Makie ~/.julia/packages/Makie/qMluh/src/makielayout/blocks.jl:371
  [5] _block
    @ ~/.julia/packages/Makie/qMluh/src/makielayout/blocks.jl:291 [inlined]
  [6] #_block#1420
    @ ~/.julia/packages/Makie/qMluh/src/makielayout/blocks.jl:259 [inlined]
  [7] _block(::Type{Axis}, ::GridPosition; kwargs::@Kwargs{limits::Tuple{Tuple{…}, Tuple{…}}, aspect::DataAspect})
    @ Makie ~/.julia/packages/Makie/qMluh/src/makielayout/blocks.jl:253
  [8] _block
    @ ~/.julia/packages/Makie/qMluh/src/makielayout/blocks.jl:246 [inlined]
  [9] #_#1418
    @ ~/.julia/packages/Makie/qMluh/src/makielayout/blocks.jl:237 [inlined]
 [10] Block
    @ ~/.julia/packages/Makie/qMluh/src/makielayout/blocks.jl:236 [inlined]
 [11] plot(pyramid::Pyramid{…}; colorbar::Bool, kwargs::@Kwargs{})
    @ PyramidScheme ~/Documents/PyramidScheme/src/PyramidScheme.jl:432
 [12] top-level scope
    @ REPL[64]:1
Some type information was truncated. Use `show(err)` to see complete types.

felixcremer · 2024-08-02T07:03:20Z

plotting of a pyramid from a provider does not work.

This is because we don't handle additional dimensions not properly, but the rgb plotting does also not work.

felixcremer · 2024-08-02T08:53:01Z

Could we also cache the Pyramid that we get from a provider?
I was playing around with it and it is very slow, because it seems as if it loads too much data.

src/pyramidprovider.jl

This will expose PyramidProvider which can then be served via HTTP so that it can be inserted into QGIS as an XYZTiles provider.

felixcremer · 2024-11-04T14:37:25Z

This kind of works now and I managed to setup a small html page which uses leaflet to show the pyramid provider, but this is quite laggy, because it throws the following error while zooming around the map:

 Error: handle_connection handler error. 
│ 
│ ===========================
│ HTTP Error message:
│ 
│ ERROR: IOError: write: broken pipe (EPIPE)
│ Stacktrace:
│   [1] uv_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64)
│     @ Base ./stream.jl:1091
│   [2] unsafe_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64)
│     @ Base ./stream.jl:1145
│   [3] unsafe_write
│     @ ~/.julia/packages/HTTP/sJD5V/src/Connections.jl:132 [inlined]
│   [4] write
│     @ ./strings/io.jl:248 [inlined]
│   [5] write
│     @ ./io.jl:794 [inlined]
│   [6] unsafe_write(http::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, p::Ptr{UInt8}, n::UInt64)
│     @ HTTP.Streams ~/.julia/packages/HTTP/sJD5V/src/Streams.jl:95
│   [7] unsafe_write
│     @ ./io.jl:803 [inlined]
│   [8] write(s::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, A::Vector{UInt8})
│     @ Base ./io.jl:837
│   [9] (::HTTP.Handlers.var"#1#2"{PyramidScheme.var"#56#57"{PyramidScheme.PyramidProvider{PyramidScheme.Pyramid{Int16, 2, Tuple{lon{DimensionalData.Dimensions.Lookups.Sampled{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, DimensionalData.Dimensions.Lookups.ForwardOrdered, DimensionalData.Dimensions.Lookups.Regular{Float64}, DimensionalData.Dimensions.Lookups.Points, DimensionalData.Dimensions.Lookups.NoMetadata}}, lat{DimensionalData.Dimensions.Lookups.Sampled{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, DimensionalData.Dimensions.Lookups.ReverseOrdered, DimensionalData.Dimensions.Lookups.Regular{Float64}, DimensionalData.Dimensions.Lookups.Points, DimensionalData.Dimensions.Lookups.NoMetadata}}}, DiskArrays.SubDiskArray{Int16, 2, Zarr.ZArray{Int16, 3, Zarr.BloscCompressor, Zarr.ConsolidatedStore{Zarr.HTTPStore}}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, false}, YAXArrays.Cubes.YAXArray{Int16, 2, DiskArrays.SubDiskArray{Int16, 2, Zarr.ZArray{Int16, 3, Zarr.BloscCompressor, Zarr.ConsolidatedStore{Zarr.HTTPStore}}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, false}, Tuple{lon{DimensionalData.Dimensions.Lookups.Sampled{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, DimensionalData.Dimensions.Lookups.ForwardOrdered, DimensionalData.Dimensions.Lookups.Regular{Float64}, DimensionalData.Dimensions.Lookups.Points, DimensionalData.Dimensions.Lookups.NoMetadata}}, lat{DimensionalData.Dimensions.Lookups.Sampled{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, DimensionalData.Dimensions.Lookups.ReverseOrdered, DimensionalData.Dimensions.Lookups.Regular{Float64}, DimensionalData.Dimensions.Lookups.Points, DimensionalData.Dimensions.Lookups.NoMetadata}}}, Dict{String, Any}}, Vector{YAXArrays.Cubes.YAXArray{Float64, 2, Zarr.ZArray{Float64, 2, Zarr.BloscCompressor, Zarr.ConsolidatedStore{Zarr.HTTPStore}}, Tuple{lon{DimensionalData.Dimensions.Lookups.Sampled{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, DimensionalData.Dimensions.Lookups.ForwardOrdered, DimensionalData.Dimensions.Lookups.Regular{Float64}, DimensionalData.Dimensions.Lookups.Points, DimensionalData.Dimensions.Lookups.NoMetadata}}, lat{DimensionalData.Dimensions.Lookups.Sampled{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, DimensionalData.Dimensions.Lookups.ReverseOrdered, DimensionalData.Dimensions.Lookups.Regular{Float64}, DimensionalData.Dimensions.Lookups.Points, DimensionalData.Dimensions.Lookups.NoMetadata}}}, Dict{String, Any}}}, Dict{Any, Any}}}}})(stream::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}})
│     @ HTTP.Handlers ~/.julia/packages/HTTP/sJD5V/src/Handlers.jl:61
│  [10] #invokelatest#2
│     @ ./essentials.jl:1055 [inlined]
│  [11] invokelatest
│     @ ./essentials.jl:1052 [inlined]
│  [12] handle_connection(f::Function, c::HTTP.Connections.Connection{Sockets.TCPSocket}, listener::HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, readtimeout::Int64, access_log::Nothing)
│     @ HTTP.Servers ~/.julia/packages/HTTP/sJD5V/src/Servers.jl:469
│  [13] (::HTTP.Servers.var"#16#17"{HTTP.Handlers.var"#1#2"{PyramidScheme.var"#56#57"{PyramidScheme.PyramidProvider{PyramidScheme.Pyramid{Int16, 2, Tuple{lon{DimensionalData.Dimensions.Lookups.Sampled{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, DimensionalData.Dimensions.Lookups.ForwardOrdered, DimensionalData.Dimensions.Lookups.Regular{Float64}, DimensionalData.Dimensions.Lookups.Points, DimensionalData.Dimensions.Lookups.NoMetadata}}, lat{DimensionalData.Dimensions.Lookups.Sampled{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, DimensionalData.Dimensions.Lookups.ReverseOrdered, DimensionalData.Dimensions.Lookups.Regular{Float64}, DimensionalData.Dimensions.Lookups.Points, DimensionalData.Dimensions.Lookups.NoMetadata}}}, DiskArrays.SubDiskArray{Int16, 2, Zarr.ZArray{Int16, 3, Zarr.BloscCompressor, Zarr.ConsolidatedStore{Zarr.HTTPStore}}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, false}, YAXArrays.Cubes.YAXArray{Int16, 2, DiskArrays.SubDiskArray{Int16, 2, Zarr.ZArray{Int16, 3, Zarr.BloscCompressor, Zarr.ConsolidatedStore{Zarr.HTTPStore}}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, false}, Tuple{lon{DimensionalData.Dimensions.Lookups.Sampled{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, DimensionalData.Dimensions.Lookups.ForwardOrdered, DimensionalData.Dimensions.Lookups.Regular{Float64}, DimensionalData.Dimensions.Lookups.Points, DimensionalData.Dimensions.Lookups.NoMetadata}}, lat{DimensionalData.Dimensions.Lookups.Sampled{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, DimensionalData.Dimensions.Lookups.ReverseOrdered, DimensionalData.Dimensions.Lookups.Regular{Float64}, DimensionalData.Dimensions.Lookups.Points, DimensionalData.Dimensions.Lookups.NoMetadata}}}, Dict{String, Any}}, Vector{YAXArrays.Cubes.YAXArray{Float64, 2, Zarr.ZArray{Float64, 2, Zarr.BloscCompressor, Zarr.ConsolidatedStore{Zarr.HTTPStore}}, Tuple{lon{DimensionalData.Dimensions.Lookups.Sampled{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, DimensionalData.Dimensions.Lookups.ForwardOrdered, DimensionalData.Dimensions.Lookups.Regular{Float64}, DimensionalData.Dimensions.Lookups.Points, DimensionalData.Dimensions.Lookups.NoMetadata}}, lat{DimensionalData.Dimensions.Lookups.Sampled{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, DimensionalData.Dimensions.Lookups.ReverseOrdered, DimensionalData.Dimensions.Lookups.Regular{Float64}, DimensionalData.Dimensions.Lookups.Points, DimensionalData.Dimensions.Lookups.NoMetadata}}}, Dict{String, Any}}}, Dict{Any, Any}}}}}, HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, Set{HTTP.Connections.Connection}, Int64, Nothing, ReentrantLock, Base.Semaphore, HTTP.Connections.Connection{Sockets.TCPSocket}})()
│     @ HTTP.Servers ~/.julia/packages/HTTP/sJD5V/src/Servers.jl:401
│   request =
│    HTTP.Messages.Request:
│    """
│    GET /8/86/131.png HTTP/1.1
│    Host: 127.0.0.1:8765
│    User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:131.0) Gecko/20100101 Firefox/131.0
│    Accept: image/avif,image/webp,image/png,image/svg+xml,image/*;q=0.8,*/*;q=0.5
│    Accept-Language: de,en;q=0.7,en-US;q=0.3
│    Accept-Encoding: gzip, deflate, br, zstd
│    DNT: 1
│    Connection: keep-alive
│    Referer: http://127.0.0.1:3000/
│    Sec-Fetch-Dest: image
│    Sec-Fetch-Mode: no-cors
│    Sec-Fetch-Site: same-site
│    Priority: u=4, i
│    
│    """
└ @ HTTP.Servers ~/.julia/packages/HTTP/sJD5V/src/Servers.jl:483

felixcremer · 2024-11-04T16:00:12Z

ToDos:
[ ] Check runtime of fetch_tile
[ ] Get rid of handle connection handler error
[ ] Make serving tiles multithreaded

meggart and others added 2 commits August 1, 2024 11:24

Add tilepyramids stuff to PyramidScheme

d811b6a

Add compat entries for new deps bump compat for YAXArrayBase

64b50d4

This comment was marked as outdated.

Sign in to view

felixcremer reviewed Aug 1, 2024

View reviewed changes

src/tilepyramids.jl Outdated Show resolved Hide resolved

felixcremer reviewed Aug 1, 2024

View reviewed changes

meggart and others added 5 commits August 1, 2024 15:18

some updates

cbf77b0

make tilesize a tuple

3713886

Fix PyramidProvider struct

e2f38fd

Revert type parameter for data_min/max

faf08f4

Fix mapTileDiskArray with tuple tilesize

f597f2f

Add test for tilepyramid

0f5ac9c

this can still get some tests for the chunking structure.

felixcremer requested changes Aug 2, 2024

View reviewed changes

src/pyramidprovider.jl Outdated Show resolved Hide resolved

felixcremer added 5 commits August 2, 2024 16:28

Add indexing test for tilepyramid

fb73e64

Add stub for pyramidprovider test

aac717b

Add compat for FixedPointNumbers

84fd29b

Make PyramidProvider work

a8e8930

This will expose PyramidProvider which can then be served via HTTP so that it can be inserted into QGIS as an XYZTiles provider.

Fix loading a Pyramid in 1.11

eaf8647

replace zero with nan and cache data

6febeca

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add tilepyramids to PyramidScheme #49

Add tilepyramids to PyramidScheme #49

meggart commented Aug 1, 2024

codecov bot commented Aug 1, 2024 •

edited

Loading

This comment was marked as outdated.

felixcremer commented Aug 1, 2024

felixcremer Aug 1, 2024

meggart Aug 1, 2024

felixcremer Aug 1, 2024

meggart Aug 1, 2024

felixcremer commented Aug 1, 2024

felixcremer commented Aug 2, 2024

felixcremer commented Aug 2, 2024

felixcremer commented Aug 2, 2024

felixcremer commented Nov 4, 2024

felixcremer commented Nov 4, 2024

Add tilepyramids to PyramidScheme #49

Are you sure you want to change the base?

Add tilepyramids to PyramidScheme #49

Conversation

meggart commented Aug 1, 2024

codecov bot commented Aug 1, 2024 • edited Loading

Codecov Report

This comment was marked as outdated.

felixcremer commented Aug 1, 2024

felixcremer Aug 1, 2024

Choose a reason for hiding this comment

meggart Aug 1, 2024

Choose a reason for hiding this comment

felixcremer Aug 1, 2024

Choose a reason for hiding this comment

meggart Aug 1, 2024

Choose a reason for hiding this comment

felixcremer commented Aug 1, 2024

felixcremer commented Aug 2, 2024

felixcremer commented Aug 2, 2024

felixcremer commented Aug 2, 2024

felixcremer commented Nov 4, 2024

felixcremer commented Nov 4, 2024

codecov bot commented Aug 1, 2024 •

edited

Loading